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(54) IMAGE PROCESSING METHOD 

(57) Abstract: 

PROBLEM TO BE SOLVED: To simply execute red-eye 
correction in a flash image with high accuracy. 
SOLUTION: An area surrounding a couple of eyes is set, 
a point between the eyes is designated (S1), a 
characteristic amount of a color at the designated point 
is calculated (S2), a skin color area is extracted from the 
inside of the setting area (S3), a mean value of the red 
chromaticity in the skin color area is calculated (S4), an 
area circumscribing the skin color area is set (S5), the 
area is divided into small areas (S6), a pupil object area 
is selected among the small division areas (S7),-a red^- 
,eye picture element in the pupil object area is extracted : 

(58) and then the red-eye picture element is corrected ~> 
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* NOTICES * 

Japan Patent Office is not responsible for any 
damages caused by the use of this translation. 

1 . This document has been translated by computer.So the translation may not reflect the original 
precisely. 

2. **** shows the word which can not be translated. 
3.1n the drawings, any words are not translated. 


CLAIMS 


[Claim(s)] 

[Claim 1] the poor color tone of the pixel by which the small field where possibility that it is 
characterized by providing the following is high was chosen as a pupil candidate field, the color 
tone extracted the poor pixel out of the pupil candidate field by which selection was carried out 
[ aforementioned ], and extraction was carried out [ aforementioned ] — an amendment — the 
image-processing method characterized by things To the photography picture by which the pupil 
was contained in the photographic subject, the field where the poor color tone of one or more 
pupils may exist is set up, the inside of the field which carried out [ aforementioned ] a setup is 
divided into two or more small fields, and it is the colour information of each aforementioned 
smallness field. The pupil in which the poor color tone exists based on positional information. 
[Claim 2] The image-processing method according to claim 1 characterized by asking for the 
central point of the field which set up and this set up the range surrounding one pair of pupils as 
a field where the poor color tone of the aforementioned pupil may exist by calculation, and 
performing division into a small field, and selection of a pupil candidate field based on the colour 
information and positional information on the basis of this central point. 

[Claim 3] It is the image-processing method according to claim 1 characterized by to perform 
division into a small field, and selection of a pupil candidate field based on the colour information 
and the positional information on the basis of the point which performed a setup of the range 
surrounding the one aforementioned pair of pupils by setting up the field of the area which 
specified one near the middle of one pair of pupils, and was beforehand set up focusing on the 
this specified point, and carried out [ aforementioned ] specification. 

[Claim 4] the field where the poor color tone of the aforementioned pupil may exist be the image 
processing method according to claim 1 characterize by perform division into a small field , and 
selection of a pupil candidate field based on the colour information and the positional information 
on the basis of the point of having specify the range in which one or more pupils surround one 
pair of pupils with a poor color tone , and having specify and this specify the point near the 
center of one pair of pupils . 

[Claim 5] It is the image-processing method of any one publication of the claim 1 characterized 
by performing division into the aforementioned smallness field by dividing into the field which 
extracted the edge pixel from the inside of the field by which a setup was carried out 
[ aforementioned ] based on colour information, and was surrounded by this edge pixel - the 
claim 4. 

[Claim 6] It is the image-processing method of any one publication of the claim 1 characterized 
by performing selection of the aforementioned pupil candidate field by searching for the 
performance index which shows pupil-likeness about each combination which made the two 
aforementioned smallness fields 1 set based on colour information and positional information, and 
choosing combination with the highest evaluation - the claim 5. 

[Claim 7] It is the image-processing method of any one publication of the claim 1 characterized 
by performing amendment of the aforementioned color tone poor pixel based on the lightness 
information on color components other than the red of a color tone poor pixel - the claim 6. 
[Claim 8] It is the image-processing method of any one publication of the claim 1 characterized 


http://www4.ipdl jpo.gojp/cgi-bin/tran_web_cgi_ejje?u=http%3A%2F%2Fwww6.ipdljp... 2003/05/20 


2/2 ^— v 


by an operator performing amendment of the aforementioned color tone poor pixel based on the 
information on the color chosen from two or more color samples - the claim 7. 
[Claim 9] Amendment of the aforementioned color tone poor pixel is the image-processing 
method of any one publication of the claim 1 characterized by enabling adjustment of the 
amendment field of a color tone poor pixel - the claim 8 by setting up arbitrarily the threshold 
about the colour information for an operator extracting a color tone poor pixel. 


[Translation done.] 
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* NOTICES * 

Japan Patent Office is not responsible for any 
damages caused by the use of this translation. 

1. This document has been translated by computer. So the translation may not reflect the original 
precisely. 

2. **** shows the word which can not be translated. 
3.1n the drawings, any words are not translated. 


DETAILED DESCRIPTION 


[Detailed Description of the Invention] 
[0001] 

[The technical field to which invention belongs] this invention relates to the amendment image- 
processing method in the poor color tone of the pupil within the photographic subject picture by 
flash plate luminescence at the time of color photography photography. 
[0002] 

[Description of the Prior Art] conventionally, the color tone of the aforementioned pupil is poor - 
- some amendment technology is proposed in the so-called bloodshot eyes (references, such as 
U.S. JP,5130789,B and JP,7-72537,A) . The technology indicated by U.S. JP.5130789.B 
surrounds the circumference of the target eye with a rectangle mask, specifies several points of 
the field of bloodshot eyes with a pointer, sets up the field of bloodshot eyes based on colour 
information, and if an object pixel is in this field, it will judge with bloodshot eyes and it will 
amend it. Under the present circumstances, the amount of amendments is changed according to 
the distance from the boundary of a field. 

[0003] Moreover, the technology indicated by JP,7-72537,A specifies the field of the target eye, 
selects a candidate pixel by colour information, also considers the information about each label 
after labeling, and evaluates whether they are bloodshot eyes. Moreover, it divides into the main 
part and periphery of bloodshot eyes about the amendment method, and the amount of 
amendments is changed. 
[0004] 

[Problem(s) to be Solved by the Invention] It is necessary to specify several [ in the field of 
bloodshot eyes ] further, and a burden starts an operator about the technology of U.S. 
JP, 51 30789, B besides surrounding the circumference of the target eye with a rectangle mask, 
although the candidate field is set up by comparing two or more sexual desire news with a 
threshold about the technology of JP.7-72537.A, since the sexual desire news of the portion 
surrounding especially bloodshot eyes changes greatly with a photography scene and individual 
differences, it is set up on the same conditions — things are difficult and become the cause by 
which this reduces yield Moreover, since it is necessary to divide a main part and a periphery 
also about the amendment method, and to change the amount of amendments, it is complicated. 
[0005] this invention was made in view of such a conventional trouble, and aims at offering the 
image-processing method which enabled it to amend bloodshot eyes with a sufficient precision 
simply. 
[0006] 

[Means for Solving the Problem] For this reason, as shown in drawing 1 as a functional block 
diagram, invention concerning a claim 1 To the photography picture by which the pupil was 
contained in the photographic subject, the field where the poor color tone of one or more pupils 
may exist is set up, and the inside of the field which carried out [ aforementioned ] a setup is 
divided into two or more small fields. The colour information of each aforementioned smallness 
field, the poor color tone of the pixel by which the small field where possibility that the pupil in 
which the poor color tone exists based on positional information is included is high was chosen 
as a pupil candidate field, the color tone extracted the poor pixel out of the pupil candidate field 
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by which selection was carried out [ aforementioned ], and extraction was carried out 
[ aforementioned ] — an amendment — it is characterized by things 

[0007] Moreover, it is characterized by for invention concerning a claim 2 asking for the central 
point of the field which set up and this set up the range surrounding one pair of pupils as a field 
where the poor color tone of the aforementioned pupil may exist by calculation, and performing 
division into a small field, and selection of a pupil candidate field based on the colour information 
and positional information on the basis of this central point, 

[0008] Moreover, it is characterized by to perform the division into a small field, and selection of 
a pupil candidate field based on the colour information and the positional information on the 
basis of the point which performed a setup of the range in which invention concerning a claim 3 
surrounds the one aforementioned pair of pupils by setting up the field of the area which 
specified one near the middle of one pair of pupils, and was beforehand set up focusing on the 
this specified point, and carried out [ aforementioned ] specification. 

[0009] moreover , the field where , as for invention concerning a claim 4 , the poor color tone of 
the aforementioned pupil may exist be characterize by perform division into a small field , and 
selection of a pupil candidate field based on the colour information and positional information on 
the basis of the point of having specify the range in which one or more pupils surround one pair 
of pupils with a poor color tone , and having specify and this specify the point near the center of 
one pair of pupils . 

[0010] Moreover, it is characterized by invention concerning a claim 5 performing division into 
the aforementioned smallness field by dividing into the field which extracted the edge pixel from 
the inside of the field by which a setup was carried out [ aforementioned ] based on colour 
information, and was surrounded by this edge pixel. Moreover, it is characterized by invention 
concerning a claim 6 performing selection of the aforementioned pupil candidate field by 
searching for the performance index which shows pupil-likeness about each combination which 
made the two aforementioned smallness fields 1 set based on colour information and positional 
information, and choosing combination with the highest evaluation. 

[001 1] Moreover, it is characterized by invention concerning a claim 7 performing amendment of 
the aforementioned color tone poor pixel based on the lightness information on color 
components other than the red of a color tone poor pixel. Moreover, it is characterized by 
invention concerning a claim 8 performing amendment of the aforementioned color tone poor 
pixel based on the information on the color which the operator chose from two or more color 
samples. 

[0012] Moreover, invention concerning a claim 9 is characterized by amendment of the 
aforementioned color tone poor pixel enabling adjustment of the amendment field of a color tone 
poor pixel by setting up arbitrarily the threshold about the colour information for an operator 
extracting a color tone poor pixel. 
[0013] 

[Effect of the Invention] the colour information out of each smallness field which divided the field 
where the poor color tone of one or more pupils may exist according to invention concerning a 
claim 1, and positional information — being based — highly precise — a pupil candidate field — 
it can choose — with, the color tone poor pixel in this pupil candidate field — an amendment — 
things can perform bloodshot-eyes amendment with a sufficient precision 

[0014] Moreover, according to invention concerning a claim 2 - a claim 4, a pupil candidate field 
can be chosen with high precision by an operator's minimum burden, moreover, the thing which is 
divided into the field surrounded by the edge pixel according to invention concerning a claim 5 — 
abbreviation — it can divide into a small field with the same color tone base, and becomes easy 
to extract a color tone poor pixel from the inside of a small field 

[0015] Moreover, according to invention concerning a claim 6, when bloodshot eyes occur 
especially on two pupils by evaluating two small fields about combination as 1 set, a pupil 
candidate field can be chosen in a high precision by the performance index using symmetric 
property, moreover, according to invention concerning a claim 7, it doubles with the inclination of 
this lightness based on color components other than red, for example, blue, and green lightness 
information — as — red or red, blue, and green lightness — an amendment — things can amend 
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to natural sensibility 

[0016] moreover — according to invention concerning a claim 8 — an operator — from a color 
sample — choosing — a favorite color — an amendment — things are made Moreover, 
according to invention concerning a claim 9, the variation in the colour information of the color 
tone poor field which embraces the grade which an operator senses that a color tone is poor, 
can expand or contract, and can set up an amendment field by setup of a threshold, and is 
generated according to a photography scene and individual differences, and its boundary region 
is absorbable. 
[0017] 

[Embodiments of the Invention] Below, the operation gestalt of this invention is explained based 
on drawing. Drawin g 2 shows the system configuration of 1 operation gestalt. The digital image 
data which read with the scanner the color picture which was photoed by the digital image data 
or the silver salt film camera of the color picture photoed by the still video camera, and was 
developed by the film are memorized by the storage 1, such as an optical disk. 
[0018] The image data memorized by the aforementioned storage 1 is read by the control unit 2, 
and image display is carried out to a monitor 3. While an operator sees the picture displayed on 
the aforementioned monitor 3, to the photographic subject picture which has produced bloodshot 
eyes by flash plate photography, by setting up a picture field as follows, the aforementioned 
control unit 2 amends by extracting the pixel of bloodshot eyes out of the this set-up field, and 
displays the amended picture on a monitor 3. 

[0019] Below, the routine of the bloodshot-eyes amendment concerning this invention is 
explained according to the flow chart below drawing 3 . Drawing JJ shows the main routine of 
bloodshot-eyes amendment. Step (it is described as S drawing.) it is the same as that of the 
following — in 1, while setting up the rectangle field surrounding the circumference of one pair of 
eyes to the photographic subject picture containing the pupil displayed on the monitor 2, it 
specifies as a reference point of the colour information which mentions one near the middle of 
the eye of a couple later, and positional information (refer to drawin g 6 ) . 

[0020] ** [ an operator may specify an object domain and a midpoint as the setting method of 
the aforementioned object domain and a midpoint, it may specify only an object domain and you 
may ask for a midpoint by calculation ] (for example, making the center of the specified rectangle 
field into a midpoint calculation) It is good also considering the area which specifies only a 
midpoint and is beforehand set up considering there as a center as an object domain. As [ both ] 
a precision of bloodshot-eyes amendment, in order that the 1st method may perform 
specification of an object domain, and specification of a midpoint by the manual, it is the highest. 

[0021] The feature of this invention is specifying the field which surrounds one pair of pupils 
even when the number of pupils with a poor color tone is one in one pair of pupils, moreover, the 
case where one pair of pupils [ both ] are bloodshot eyes — 1 time of a block definition — 
simultaneous — two bloodshot eyes — an amendment — things are made For example, with the 
technology indicated by aforementioned U.S. JP.51 30789, B, in order to carry out a block 
definition at a time about one pupil, an operator's degree of intervention becomes large. 
[0022] The colour information of the point specified at the aforementioned step 1 in Step 2, for 
example, R, (red) G (green) B (blue) Total value of each lightness value (R+G+B) It calculates as 
characteristic quantity. At Step 3, the field of this beige pixel is extracted by making the pixel 
within the threshold specified by the difference of characteristic quantity with the characteristic 
quantity of the aforementioned specifying point into a beige pixel. That is, since the point near 
the middle of one pair of eyes is located in the root portion of a nose by the aforementioned 
specifying point and it is presumed that the color in the point is beige, the field which is carrying 
out the color near this point can be presumed to be a beige pixel. In addition, the hue and 
saturation which are general colour information are not cared about as characteristic quantity. 
Thus, labeling of the pixel within the aforementioned threshold is carried out, and the labeling 
field where a specifying point is included is extracted as a beige field. 
[0023] At Step 4, the average Rskin of the chromaticity [=R/(R+G+B)] of the red in the 
aforementioned beige field is calculated. Rectangle field circumscribed to the beige field 
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extracted at the aforementioned step 3 in order to reduce the following amounts of operations, 
and the amount of data required for this operation at Step 5 (both eyes the portion between 
eyes included so that it may circumscribe simultaneously one oblong field) It sets up as a new 
object domain. A setup of this rectangle field is performed by being automatic. Although you may 
carry out manually, it is automatic, and when the operator has performed a latus block definition 
in which fields other than a face are included, it can respond by carrying out. 
[0024] The aforementioned object domain is divided into a small field at Step 6. The sub routine 
of this small field division is explained according to the flow chart of draw ing 4 . At Step 1 1, it 
asks for an edge pixel in the field by which a setup was carried out [ aforementioned ]. How to 
ask for this edge pixel is lightness. (R+G+B) It considers as characteristic quantity and asks for 
the sum of the absolute value of the difference of 4 pixels [ of circumferences of an object 
pixel ] lightness, and if it is more than the threshold that this set up, an object pixel will be 
judged to be an edge pixel. In addition, you may use edge-detection filters, such as a Sobel 
operator and a Prewitt operator. 

[0025] At Step 12, labeling of the edge pixel detected at the aforementioned step 1 1 is carried 
out. At Step 13, the pixel which had the methods of three surrounded at least by the edge pixel 
of the same label is changed into the same label as the aforementioned edge pixel. That is, the 
field surrounded with the edge serves as the same label. Thus, a setting field is divided into a 
small field with the same label. 

[0026] It returns to the main routine of drawin g 3 , and the field where possibility that the pupil 
in which a poor color tone exists is contained is high is chosen as a pupil candidate field at Step 
7 from the small fields by which division was carried out [ aforementioned ]. The sub routine of 
pupil candidate field selection of the aforementioned step 7 is explained according to the flow 
chart of drawjng 5 . 

[0027] With this operation gestalt, it asks for pupil-likeness by calculating the following 
characteristic quantity about all combination, using as 1 set two of two or more small fields by 
which division was carried out [ aforementioned ]. Step 21 — the chromaticity [=R/( R+G+B)] of 
the red of each smallness field — asking — the inside of a field — the highest value of a red 
chromaticity (Ri) and the time of making into a zero the point near [ which was specified at the 
aforementioned step 1 ] the middle of an eye and an eye — the aforementioned highest value 
(Ri) Position on the X-axis of a pixel, and a Y-axis (Xi, Yi) It considers as characteristic quantity 
Here, i expresses the value which discriminates a small field. 

[0028] For example, when combining the small field 1 and the small field 2, a pupil candidate field 
is chosen as follows. At Step 22, the characteristic quantity C12 of colour information and the 
characteristic quantity P12 of positional information are calculated by the following formula, 
characteristic quantity P12=X1+X2+Y1+Y2 of characteristic quantity C12=R1+R2 positional 
information of colour information — here Since pupil-likeness is so high that the characteristic 
quantity of positional information is so small that the characteristic quantity of colour 
information is large, at Step 23 The characteristic quantity E12 which shows pupil-likeness is set 
up with E12=C12-P12, and the color tone of at least one side is poor in the combination of the 
small field where this value is the highest, (bloodshot eyes) It extracts as a pupil candidate field 
where possibility that the pupil which has become is included is high. 

[0029] Here, to the point between eyes, when bloodshot eyes are in both pupils, the thing with 
high pupil-likeness will depend right-and-left one X1 on a bird clapper small, if X2 of + and 
another side is totaled as -, so that the characteristic quantity of positional information becomes 
small. Moreover, when the combination of both pupils is chosen, the thing as which the case 
where only one pupils are bloodshot eyes is sufficient will be because it becomes large [ the 
characteristic quantity of the color upper part ], if that the value of positional information P12 
becomes small even if there are no bloodshot eyes in one pupil, and there are a bloodshot eyes 
also compares it with the combination in which bloodshot eyes do not have one. 
[0030] The color tone out of the pupil candidate field which returned to the main routine of 
drawing 3 and was extracted at Step 7 in Step 8 is poor, (bloodshot eyes) A pixel is extracted. 
This extraction method is as follows, the average Rskin of the beige field for which the 
chromaticity [=R/(R+G+B)] of the red of an object pixel asked at Step 2 — large — in addition - 
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- and characteristic quantity of the colour information of an object domain (Ri) from — when 
larger than the value which lengthened the set-up threshold TH, it considers as a bloodshot- 
eyes pixel That is, let the pixel which fills the following formula be a bloodshot-eyes pixel. 
[0031] At the R/(R+G+B) >Rskin**R/(R+G+B) >Ri-TH step 9, the bloodshot-eyes pixel 
extracted at Step 8 is amended to a proper color tone. In this case, since ail pixels are looked 
very unnaturally to the same pixel value with an amendment, lightness information other than red 
is harnessed, for example, digital value — R, G, and B — the amendment of the gray color which 
digital value doubled with the low value most when it was R= 200, G= 50, and B= 30, respectively 
In this case, since the lightness of R=G=B=30 can be amended in the pupil picture reflected to 
some extent, it is visible to natural sensibility. 

[0032] since [ or ] it corresponds when the color of a pupil is not a gray from the first — for 
example, the above — - the same — digital value — R, G, and B — when it is R= 200, G= 50, and 
B= 30, respectively, G and B maintain a value as it is, and double R with 50 of G by the value of 
the larger one of G and B, and this case In addition, you may make it double the value of R with 
40 by the average of G and B, and this case. 

[0033] furthermore, the case where the color of an operator's favorite pupil is not suited — 
pallet of the color of a pupil (sample) from — a favorite color — choosing — I have you — a 
basis [ news / sexual desire / the ] — amendment As the method of amendment, the following 
procedures perform, for example. It asks for the colour information of the selected color, for 
example, YIQ, by the following formula. 
5034] 

^Equation 1] 


Y 


0.3 0.59 

an 


R 

1 


0.6 -0.28 

-0.32 


G 

Q 


0.21 -0.52 

0.31 


B 


[0035] In a bloodshot-eyes pixel, the maximum of the value of B+G is calculated and this is set 
to Lmax. To the lightness Y of a pallet, it is L (=B+G) of an object pixel. The value broken by 
Lmax is multiplied, (refer to the following formula) . 

Maintaining, the value of Y -YxL/LmaxI and Q is transformed inversely to R, G, and B, and makes 
this digital value correction value. By doing in this way, it is more visible to nature. 
[0036] That is, about lightness, although a pallet performs selection of a color, since amendment 
is performed taking advantage of the lightness information on components other than the red of 
a bloodshot-eyes pixel, natural sensibility can also be acquired. In a series of processings, 
although it asked for the red chromaticity, you may use combination, such as other general 
colour information, for example, lightness, a hue, and saturation. According. to this operation 
gestalt, the following effects are acquired. 

[0037] ** There are few degrees of intervention of an operator and they end. 

** the influence according to a photography scene and individual differences as compared with 

the method of dividing only using colour information, since field division is carried out using edge 

information — receiving — hard — high yield — an amendment — things are made 

** When the color of the above and a pupil is not a gray, also in other than a Japanese, it can 

respond by performing the aforementioned amendment. 

[0038] Next, the 2nd operation gestalt is explained. With the operation gestalt of the above 1st, a 
part of bloodshot-eyes portion to correct according to the individual differences of a bloodshot- 
eyes portion may be amended, or it may be amended to the portion which is not bloodshot eyes. 
In order to prevent this, while an operator looks at a picture, the threshold for distinguishing the 
existence of amendment is adjusted and it enables it to adjust an amendment field with this 
operation gestalt. 

[0039] An operator specifically looks at a picture, when an operator judges repeatedly that it is 
amended to the portion which is not bloodshot eyes conversely until it lowers the value of the 
threshold TH set to the aforementioned bloodshot-eyes pixel extraction and the amendment 
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field was expanded appropriately, when it was judged that a part of portion of bloodshot eyes is 
amended, the value of the aforementioned threshold TH is raised, and it repeats until an 
amendment field is reduced appropriately. 

[0040] There is also a portion undertaken to subjectivity about a field setup of bloodshot eyes, 
and amendment to which an operator's subjectivity is satisfied to do in this way can be 
performed. Moreover, the variation in the colour information of the bloodshot-eyes [ which are 
generated according to a scene and individual differences ] and bloodshot-eyes circumference is 
absorbable. 


[Translation done.] 
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* NOTICES * 

Japan Patent Office is not responsible for any 
damages caused by the use of this translation. 

1 This document has been translated by computer. So the translation may not reflect the original 
precisely. 

2.**** shows the word which can not be translated. 
3.1n the drawings, any words are not translated. 


DESCRIPTION OF DRAWINGS 


[Brief Description of the Drawings] 

[ Drawin g 1] The block diagram showing the composition and the function of this invention. 
[Drawin g 2] Drawing showing the system configuration of 1 operation gestalt. 
[Drawing 3] The flow chart which shows the main routine of the bloodshot-eyes amendment in 
an operation gestalt same as the above. 

[D rawin g 4] The flow chart which shows the sub routine of small field division of a routine same 
as the above. 

[Drawing 5] The flow chart which shows the sub routine of pupil candidate field selection of a 
routine same as the above. 

[Drawing 6] Drawing showing the specification method of the object domain in an operation 
gestalt same as the above. 
[Description of Notations] 

1 Storage 

2 Control Unit 

3 Monitor 


[Translation done.] 
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[ Drawing 6] 



[Translation done.] 
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